Periodical generation of network measurement data

ABSTRACT

Some embodiments provide a method that generates different network measurements data (e.g., network topology, bandwidth estimation of different paths, etc.) for a pair of endpoints upon receiving a network administrative request (e.g., an application programming interface (API) request) or by other means (e.g., automatically and without intervention of a user). In some embodiments, the method is implemented by a network measurement agent operating on each endpoint and a centralized service component (e.g., web service layer) executing on a network manager machine (e.g., a controller) that responds to measurement requests. The network measurement agent probes the network periodically (i.e., in certain time intervals) or upon an API request that it receives through the network manager machine to gather the measurement data of a particular network topology between a pair of endpoints.

BACKGROUND

Network fabric's measurement tools have been widely explored over thepast several years. The most important characteristics of a networkfabric are network topology, bandwidth, and latency. Measurement of suchcharacteristics of a network is important for network troubleshootingand optimization of network applications such as end-to-end transportperformance, intelligent overlay network routing, and peer-to-peer filedistribution.

There are quite a few tools that are currently used to estimatebandwidths of hop-by-hop links or end-to-end paths in a network.However, with recent evolution in network technology, network topologieshave become more and more complex. For example, in recent years,Equal-Cost Multi-Path (ECMP) Internet Protocol (IP) routing has beenwidely deployed to implement load balancing in the networks. The ECMProuting potentially offers substantial increases in bandwidth of thenetwork by load-balancing the network traffic over multiple paths. Forsuch networks, the current measurement tools can only estimate thebandwidth for a single random path.

There are also multiple paths discovery tools that are currently used todetect multiple paths between a given Internet Protocol (IP) endpointspair in networks that deploy load balancing forwarding elements (e.g.,load balancing routers). These tools, however, have their ownshortcomings. For example, most available network path discovery tools(e.g., traceroute) can only discover a limited number of paths (e.g.,three paths) in best-case scenario. Additionally, there is always thepossibility that these tools fail to discover true nodes and links in anetwork or worse yet, introduce false links that in reality don't existin the network.

For data center networks, the aforementioned network characteristics arerarely explored and those network fabrics are mostly treated ashigh-capacity black boxes. However, with the evolution ofSoftware-Defined Data Centers (SDDC), more and more aspects of theinfrastructure of the network are abstracted and built on top of thenetwork fabric, and understanding network fabric characteristics hasbecome crucial for planning, monitoring, and troubleshooting of thenetworks.

SUMMARY

Some embodiments of the invention provide a novel method for measuringnetwork characteristics (e.g., bandwidth, path diversity, etc.) withoutrequiring administrative access to intermediate network elements andindependent from the specific network fabric. In some embodiments, themethod uses a multi-path estimation technique that takes into accountall the different paths between a given pair of endpoints. The method ofsome embodiments measures the bandwidth (e.g., available bandwidth,capacity, etc.) between a given pair of endpoints (e.g., a sourceforwarding element and a destination forwarding element at the edges ofthe network) by identifying the routing paths between the endpoints andcalculating the bandwidth for each path in order to estimate the networkbandwidth between the endpoints.

The method of some embodiments identifies a network topology that isinvolved in routing of the network traffic between the endpoints. Theidentified topology covers (1) all the routing paths (e.g., hardware andsoftware-defined paths) between the endpoints and (2) all the forwardingelements (e.g., hardware and software-defined non-edge forwardingelements) along each path that forwards the network traffic (e.g.,network packets). The method identifies, and groups together, thereceiving interfaces as well as the forwarding interfaces of eachforwarding element along different paths between the endpoints. By doingso, the method can identify a real link between a pair of forwardingelements when a set of forwarding interfaces of each one of theforwarding elements in the pair points to the other forwarding element.

After identifying the network topology, including all of the existingrouting paths between a source endpoint and a destination endpoint(e.g., in a software-defined datacenter), and all of the forwardingelements along each routing path, the method of some embodiments choosesa single flow signature for each identified path. The method generatesbandwidth estimation probing packets with each flow signature (e.g.,having a different transmission control protocol (TCP) or user datagramprotocol (UDP) source port) in order to calculate an estimated bandwidthfor each link (e.g., hop-by-hop link) of the path corresponding to theflow signature.

The method of some embodiments aggregates all of the calculatedestimated bandwidths of the links in order to estimate the bandwidthbetween the two endpoints. In order to aggregate, the method firstnormalizes the estimated bandwidths of diverged links. That is, themethod of some embodiments calculates a minimum bandwidth between all ofthe links that share the same starting and ending forwarding elements inorder to calculate the bandwidth between the shared starting and endingforwarding elements. The method then calculates a minimum bandwidthbetween the shared forwarding elements of the diverged links and allother non-diverged links that lie between the endpoints (i.e., thesource endpoint and the destination endpoint) in order to estimate thebandwidth between the two endpoints.

In some embodiments, the method generates different network measurementsdata (e.g., network topology, bandwidth estimation of different paths,etc.) for a pair of endpoints upon receiving a network administrativerequest (e.g., an application programming interface (API) request) or byother means (e.g., automatically and without intervention of a user). Insome embodiments, the method is implemented by a network measurementagent operating on each endpoint and a centralized service component(e.g., web service layer) executing on a network manager machine (e.g.,a controller) that responds to measurement requests. The networkmeasurement agent of some embodiments probes the network periodically(i.e., in certain time intervals) or upon an API request that itreceives through the network manager machine to gather the measurementdata of a particular network topology between a pair of endpoints (e.g.,the pair of endpoints on which the network measurement agents operate).

In some embodiments, when a network measurement feature is turned “on”or enabled (e.g., through an API), the network measurement agent canprovide (e.g., to a network measurement server or the network managermachine) real-time data (e.g., bandwidth capacity, available bandwidth,etc.) or the most recent measurement data on each possible path (e.g., alink, multiple links, etc.) between a given pair of internet protocol(IP) endpoints.

For instance, with an API request via a network measurement interface,the network measurement agent of some embodiments can periodicallymeasure the bandwidths between a pair of endpoints of the network. Thisis important because the data (e.g., real-time data) can be used in anumber of different ways, including forwarding traffic along differentpaths. For instance, the always “on” feature can be used to retrievereal-time measurement data and assign different flow signatures orhashes for different data flows depending on the measurement data. Withthe hop-by-hop measurements and the different flow signatures associatedwith a given topology, the flow of data, from a source to a destination,can even be controlled on a per hop basis.

The network measurement server of some embodiments stores the providednetwork measurement data in a storage (e.g., a database). Rather thansimply storing the measurement data, the server of some embodimentsaccumulates or aggregates the measurement data. That is, for a pair ofendpoints, the server stores the measurement data and then updates thedata with each updated measurement data it receives from the measurementagents. By knowing the estimated bandwidth for different paths, anetwork controller that controls a forwarding element in the routingpath can decide or specify which path to use for packets that areassociated with a particular data flow (e.g., elephant flows, mouseflows, etc.). In this manner, one set of packets belonging to aparticular data flow (e.g., mouse flow) is not held behind a routingpath that is used by another set of packets belonging to another dataflow (elephant flow) for a prolonged period of time.

Alternatively, or conjunctively, the network measurement agent of someembodiments provides the network measurement data to a transport node(e.g., edge forwarding element) to facilitate monitoring andoptimization of the routing of network data flows by the transport node.For instance, the forwarding element can decide to switch thetransportation of a particular data flow from one path to another pathbased on the capacity and available bandwidth of each path. In someembodiments, a forwarding element (e.g., an edge forwarding element)breaks an elephant flow to multiple mice flows and route each mouse flowto a different routing path based on the capacity and availablebandwidth that it receives from its corresponding network measurementagent. The forwarding elements of some embodiments make such decisionsthrough one or more controllers (e.g., hardware or software-definedcontrollers) that control the forwarding elements.

The preceding Summary is intended to serve as a brief introduction tosome embodiments as described herein. It is not meant to be anintroduction or overview of all subject matter disclosed in thisdocument. The Detailed Description that follows and the Drawings thatare referred to in the Detailed Description will further describe theembodiments described in the Summary as well as other embodiments.Accordingly, to understand all the embodiments described by thisdocument, a full review of the Summary, Detailed Description and theDrawings is needed. Moreover, the claimed subject matters are not to belimited by the illustrative details in the Summary, Detailed Descriptionand the Drawings, but rather are to be defined by the appended claims,because the claimed subject matters can be embodied in other specificforms without departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a process that some embodiments use toestimate the bandwidth between two endpoints.

FIG. 2 illustrates an example of how the method of some embodimentsestimates the bandwidth between a pair of endpoints.

FIG. 3 conceptually illustrates a process of some embodiments thatidentifies a network topology between two endpoints.

FIG. 4 illustrates an example of how the process of some embodimentsidentifies the network topology in some embodiments.

FIG. 5A-5C illustrate an example of how the method of some embodimentsidentifies different flow signatures for different paths between a pairof endpoints.

FIG. 6 illustrates an example of choosing a single flow signature foreach identified path among all the flow ids assigned to that path.

FIG. 7 conceptually illustrates a process of some embodiments thatcalculates an estimated bandwidth for each link that is along thedifferent routing paths between a pair of endpoints.

FIG. 8 illustrates an example of how the process of some embodimentscalculates an estimated bandwidth for each link along different paths.

FIG. 9 conceptually illustrates a process of some embodiments thataggregates the calculated bandwidths of all the links between the pairof endpoints to generate an estimated bandwidth between the endpoints.

FIG. 10 illustrates an example of how the process aggregates theestimated bandwidths of the links.

FIG. 11 presents a state diagram that illustrates example operationsperformed by such a network measurement agent.

FIG. 12 illustrates an example of how the method of some embodimentsroutes different sets of packets that belong to different data flowsthrough different paths between two endpoints based on the estimatedbandwidth of each path.

FIG. 13 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments of the invention provide a novel method for measuringnetwork characteristics (e.g., bandwidth, path diversity, etc.) withoutrequiring administrative access to intermediate network elements andindependent from the specific network fabric. In some embodiments, themethod uses a multi-path estimation technique that takes into accountall the different paths between a given pair of endpoints. The method ofsome embodiments measures the bandwidth (e.g., available bandwidth,capacity, etc.) between a given pair of endpoints (e.g., a sourceforwarding element and a destination forwarding element at the edges ofthe network) by identifying the routing paths between the endpoints andcalculating the bandwidth for each path in order to estimate the networkbandwidth between the endpoints.

In some embodiments, the method estimates the bandwidth between sourceand destination endpoints. In some embodiments, the source ordestination endpoint can be any machine (e.g., virtual machine (VM),computing device). In some embodiments, the endpoint can also be an endnode forwarding element. In some embodiments, the end node forwardingelements are edge forwarding elements (EFEs), which are the firstforwarding elements to which the source machine and destination machinesrespectively send and receive the packets.

The edge forwarding elements of some embodiments are software-definedforwarding elements that execute on a host computing device, on whichthe source and destination VMs execute. For instance, the source machinecan be the VM that sends the packets, and the destination machine can bethe VM that receives the packets. In some embodiments, the source VM ison one host computing device, on which the source edge forwardingelement executes, while the destination VM is on another host computingdevice, on which the destination edge forwarding element executes.

FIG. 1 conceptually illustrates a process 100 that some embodiments useto estimate the bandwidth between two endpoints (e.g., two edgeforwarding elements). In some embodiments the process 100 is implementedby a bandwidth measurement agent or tool on each transport node (e.g.,source or destination endpoint) to measure and combine network fabricmeasurements including topologies, network bandwidth estimations.

The process 100 begins by receiving (at 110) a pair of endpoints (e.g.,internet protocol (IP) pair of endpoints) and generating different flowsignatures for different routing paths between the endpoints. In someembodiments, the process 100 identifies the different paths and all thenon-edge forwarding elements (NEFEs) along each identified path using anenhanced traceroute algorithm (e.g., enhanced Paris traceroutealgorithm). Identifying the different paths and the NEFEs along thepaths is discussed in more detail in Section I below, while assigningflow signatures to each identified path is discussed in more detail inSection II below.

The process 100 then selects (at 120) a first flow signature that it hasassigned to a first path between the endpoints. At 130, the process 100calculates an estimated bandwidth for each link (hop-by-hop link) of thefirst path. In some embodiments, the process forwards probing packetswith the flow signature by a bandwidth estimation tool to the path tocalculate the estimated bandwidth of each link on the path. More detailon forwarding probing packets to different paths by the bandwidthestimation tool is discussed in Section II below.

The process 100 then determines (at 140) whether there are more flowsignatures assigned to more paths or not. If the process determines thatthere are more flow signatures, the process 100 returns to 120 to selectthe next flow signature and calculate the bandwidths of all the links ona path corresponding to the next flow signature. On the other hand, ifthe process 100 determines (at 140) that there are no more flowsignatures left, the process 100 proceeds to 150.

At 150, the process aggregates all calculated bandwidths of all routingpaths. The process of some embodiments aggregates the estimatedbandwidths by first normalizing all diverged links that share the samestarting and ending nodes (i.e., NEFEs) recursively and then calculatesthe minimum bandwidth between the shared forwarding elements of thediverged links and all other non-diverged links that lie between theendpoints. More detail on aggregating the estimated bandwidth ofdifferent links is also discussed in Section II below. The process 100then estimates (at 160) the bandwidth between the two endpoints based onthe aggregated estimated bandwidths. The process 100 then ends.

Some embodiments perform variations on the process 100. The specificoperations of the process 100 may not be performed in the exact ordershown and described. The specific operations may not be performed in onecontinuous series of operations, and different specific operations maybe performed in different embodiments.

FIG. 2 illustrates an example of how the method of some embodimentsestimates the bandwidth between a pair of endpoints. More specifically,FIG. 2 illustrates a network topology 200 that includes a pair ofendpoints 205 and 230 with four forwarding elements 210-225 (e.g.,routers, switches) between them. The pair of endpoints in someembodiments includes a source and a destination at the edges of twodifferent hosts.

Each of the forwarding elements (EFEs and NEFEs) in a network (e.g., adatacenter network) of some embodiments can be a physical forwardingelement (PFE) or a logical forwarding element (LFE). A physicalforwarding element, in some embodiments, can be categorized as asoftware-defined forwarding element or a hardware forwarding element. Asoftware-defined forwarding element such as Open vSwitch is a softwareimplemented switch that operates on a computing device (e.g., x86 box).A hardware forwarding element, on the other hand, is hardware designedto support in hardware forwarding. For instance, the forwarding elementmay have application-specific integrated circuits (ASICs) with a ContentAddressable Memory (CAM) or Ternary CAM (TCAM). In general each of thesePFEs can be a switch, a router, or any other appliance that is capableof receiving and forwarding packets.

Returning to FIG. 2, this figure also shows that the topology estimationmethod has identified two routing paths between the source endpoint 205and the destination endpoint 230. The first identified path in thefigure includes the nodes Src, L, A, C, and Dst while the secondidentified path includes the nodes Src, L, B, C, and Dst. The method hassent probing packets with an estimation tool to estimate the bandwidthof all the links on each path. These bandwidth estimations areillustrated as B(Src, L) which is the bandwidth estimation for the linkbetween node 205 and node 210, B(L, A) which is the bandwidth estimationfor the link between node 210 and node 215, B(L, B) which is thebandwidth estimation for the link between node 210 and node 220, B(A, C)which is the bandwidth estimation for the link between node 215 and node225, B(B, C) which is the bandwidth estimation for the link between node220 and node 225, and finally B(C, Dst) which is the bandwidthestimation for the link between node 225 and node 230.

While the embodiments described above and below provide a method forestimating bandwidth (hop-by-hop links and end-to-end paths) for a pairof endpoints, one of ordinary skill in the art will realize that manyembodiments of the invention are equally applicable to estimatinglatency (hop-by-hop links and end-to-end paths) for a pair of endpoints.In other words, the methodologies and architecture described above andbelow are used in some embodiments to estimate both bandwidth andlatency of the different paths and links between a pair of endpoints.

The method of some embodiments then aggregates the calculated bandwidthsby normalizing the diverged links and calculating a minimum of all thebandwidth estimations of the normalized diverged links and all thebandwidth estimations of the non-diverged links to generate thebandwidth estimation between the endpoints 205 and 230. As describedabove, diverged links are links that share the same starting and endingnodes (diverged nodes). In the illustrated example, the diverged linksare links LA and AC on the first path and links LB and BC on the secondpath. The combination of these links shares the same starting node L aswell as the same ending node C.

In order to normalize the diverged links and calculate the bandwidthestimation between the shared nodes of the diverged links, the method ofsome embodiments first calculates the minimum bandwidth of the divergedlinks on the first path and adds the result to the minimum bandwidth ofthe diverged links on the second path. Therefore in FIG. 2 the estimatedbandwidth between the diverged nodes L and C (i.e., B(L, C)) will be theminimum bandwidth of the diverged links LA and AC (i.e., MIN(B(L, A),B(A, C))) plus the minimum bandwidth of the diverged links LB and BC(i.e., MIN(B(L, B), B(B, C))).

In order to complete the aggregation, the method of some embodimentsafter calculating the bandwidth estimation between the diverged nodes,calculates the minimum bandwidth between the diverged nodes that areshared by the diverged links and all other non-diverged links that liebetween the endpoints. The result of this calculation will be thebandwidth estimation between the two endpoints 205 and 230.

Having described a brief overview of a bandwidth measurement methodbetween a pair of endpoints, several different aspects of the method nowwill be described in detail in the following sections. In particular,Section I describes an enhanced traceroute method that provides anetwork topology between a pair of endpoints. This is followed bySection II that describes how the method of some embodiments estimatesthe bandwidth between a pair of endpoints using the network topology andflow signatures. Section III then describes a network measurement agentthat periodically probes a network. Section IV then describes anelectronic system for implementing some embodiments of the invention.

I. Network Topology

The method of some embodiments identifies a network topology that isinvolved in routing of the network traffic between a pair of endpoints.The network traffic, in some embodiments, includes network packets thatbelong to different data flows as well as individual network packets.The term “packet” is used here as well as throughout this application torefer to a collection of bits in a particular format sent across anetwork. One of ordinary skill in the art will recognize that the term“packet” may be used herein to refer to various formatted collections ofbits that may be sent across a network, such as Ethernet frames, TCPsegments, UDP datagrams, IP packets, etc.

The identified topology covers (1) all the routing paths (e.g., hardwareand software-defined paths) between the endpoints and (2) all theforwarding elements (e.g., hardware and software-defined non-edgeforwarding elements) along each path that forwards the network traffic(e.g., network packets). The method identifies, and groups together, thereceiving interfaces as well as the forwarding interfaces of eachidentified forwarding element along different paths between theendpoints. By doing so, the method can identify a real link between apair of forwarding elements when a set of forwarding interfaces of eachone of the forwarding elements in the pair points to the otherforwarding element.

Traditionally, different traceroute algorithms were used to discover anetwork topology (i.e. different paths and different forwarding elementsalong the paths) between two endpoints. However, after introduction ofEqual-Cost Multi-Path (ECMP) Internet Protocol (IP) routing mechanismand load balancing in the networks where there is no more a single routefrom a source to a destination, the traditional traceroute algorithmsfail to discover true nodes and links between the nodes. These problemsarise because traditional traceroute methods discover hops along a routewith a series of probing packets while a load balancing router maydirect the probing packets along different paths.

The Paris traceroute algorithm was introduced to solve the multipathdiscovery shortcomings of classic traceroute algorithms. However, Paristraceroute only returns forwarding elements' receiving interfaces andtherefore can generate false measurements in ECMP fabrics with routersconfigured with numbered interfaces, as internet control messageprotocol (ICMP) reply messages can be returned on asymmetric paths whichresults in the prober deducing incorrect interfaces at which probepackets have arrived. Moreover, Paris traceroute only discoverinterfaces and lacks the representation for routers with multipleinterfaces in the network topology.

Unlike traditional traceroute algorithms or Paris traceroute, someembodiments provide a novel method that discovers both forwardinginterfaces and receiving interfaces of the forwarding elements of amulti-path network and groups the discovered interfaces together to (1)identify each physical forwarding element (e.g., software-defined orhardware forwarding element) separately and (2) identify whichforwarding elements point to each other to draw a link between them. Inorder to do so, the method of some embodiments uses the “record route”option in the probing packets' headers to accurately identify theforwarding elements and the links between them. More specifically, theinitiator of the probing packet, upon receiving the packet back from aforwarding element with its corresponding ICMP error message, parses therecord route option in the inner packet header and the tip entry in theheader will be the forwarding interface of the forwarding element thathas returned the packet with the error message.

FIG. 3 conceptually illustrates a process 300 of some embodiments thatidentifies a network topology between two endpoints (e.g., two transportnodes). In some embodiments, the process 300 is initiated using APIcalls from a network measurement server via the network measurementinterface to the network measurement agents that operate on eachtransport node (e.g., source and destination nodes). FIG. 3 will bedescribed by reference to FIG. 4 which illustrates an example of how theprocess 300 identifies the network topology in some embodiments.

The process 300 begins by initiating (at 310) the time to live (TTL)value in the probing packet's header and routing the probing packet fromthe source node to the next forwarding element. The TTL is a value in apacket's header that tells a forwarding element whether or not thepacket has been in the network for too long and if it has, the packetshould be discarded. For a number of reasons, packets may not getdelivered to their destination in a reasonable length of time. Forexample, a combination of incorrect routing tables could cause a packetto loop endlessly. A solution is to discard the packet after a certaintime and send a message to the originator, which can decide whether toresend the packet or drop it.

Each time the packet is received by a forwarding element, the forwardingelement decreases one unit from the TTL value and when the packet's TTLvalue reaches zero, it will be discarded and returned to the originatorof the packet by the forwarding element with an error message. Forinstance when a TTL value is set to 3 and the packet reaches the thirdforwarding element on its route, the TTL value is decremented to zerowhich indicates to the forwarding element that the packet has beenforwarded in the network for too long and subsequently the forwardingelement returns the packet with an error message (e.g., ICMP errormessage) to its originator (e.g., the source node).

The process 300 then forwards (at 320) the probing packets to each node(i.e., each NEFE) on TTL level and identifies all forwarding andreceiving interfaces of each individual node on that TTL level. Suchforwarding of probing packets and identification of interfaces of theintermediate nodes are described in more detail below by reference toFIG. 4.

FIG. 4 shows a source node 405 and a destination node 440 and severalNEFEs 410-435 along two different paths between the pair of nodes (i.e.pair of endpoints). This figure also shows the network manager 450 whichin some embodiments is used to manage and configure the transport nodes405 and 410. The network manager 450 of some embodiments includesapplication programming interfaces (APIs) 455 to communicate with thenetwork measurement agents 460 that execute on the transport nodes 405and 410. The network measurement agents of some embodiments execute ontransport nodes at the edge of the network including a hypervisor of ahost that includes several virtual machines. The network managementserver provides a centralized service component (e.g., web servicelayer) that serves the network measurement requests.

Through the measurement interface 455, the network manager 450 interactswith network measurement agents on different transport nodes at the edgeof the network for collecting, aggregating, and combining networkmeasurements (e.g., bandwidth, latency, topology, etc.) and making thosemeasurements available to other network elements and/or services. Also,the network manager 450 may perform additional management services bysending (e.g., pushing) management data (e.g., bridge information,virtual interface information, tunnel information, etc.) to the physicalforwarding elements (e.g., transport nodes). The network manager of someembodiments performs the configuration by sending (e.g., pushing) dataflows to the transport nodes. Instead of sending flows, the networkmanager 450 of some embodiments exchanges forwarding state informationthat is translated into a set of data flows at the transport nodes.Several examples of such additional services will be discussed in moredetail below in Section III by reference to FIG. 12.

In the example of FIG. 4, the measurement interface 455 is shown asbeing implemented on the same computing device as the network manager450. However, the measurement interface can be implemented on anymachine, virtual or dedicated. As an example, the measurement interfacecan be implemented on a hypervisor, a service virtual machine (VM), aseparate measurement aggregating machine, etc. The measurement interfacecan also be implemented, in some embodiments, on a same device orhypervisor as a software or virtual forwarding element (e.g., thetransport nodes 405 and 410). The measurement interface can be anapplication or a plug-in component that runs on a particular operatingsystem.

In order to recognize the network topology between the pair ofendpoints, the source node 405 forwards the first probing packet (orfirst set of probing packets) to the next forwarding element 410 (i.e.,node L) with a TTL value set to 1. Node L receives the probing packetand inserts the forwarding interface “1” in the probing packet's IPrecord route header, Node L then decreases the TTL value by one, andthis reduction in turn tells node 410 that the packet has been inrouting for too long since the TTL value will be set to zero.Subsequently, node L generates a Time-to-live exceeded ICMP messagewhich copies the probing packet IP header (also including record routeheader) and the first 8 bytes of the probing packet's IP payload as ICMPmessage payload, then sends back to the source EFE 405 via the receivinginterface “0”.

One of ordinary skill in the art would realize that the number ofinterfaces associated with each forwarding element in reality can bemany more than one or two interfaces that are shown in the figure andthe only one or two interfaces illustrated in the example figure are forsimplicity of description. Also, one of ordinary skill in the art wouldrealize that the actual addresses for the interfaces are not simpledigits (i.e., “0,” “1,” “2,” etc.) and these digits are used only asexemplary figures.

After receiving the probing packet back, as discussed above, the sourcenode 405 parses the receiving interface “0” and the forwarding interface“1” from their corresponding places in the ICMP message. The processthen generates more new probing packets with different flow signaturesto discover all other sibling nodes of node L that are on the same TTLlevel. In the illustrated example there is no neighboring node at thesame TTL level as node L is.

Returning to FIG. 3 the process 300 then determines (at 330) whether allnodes on different paths are probed. If the process determines that ithas reached the destination node (i.e., transport node 440) the process300 proceeds to 350 (which will be described below), otherwise theprocess increases the TTL value by one (at 340) and returns to 320 whichwas described above.

In FIG. 4, the source node then routes another probing packet (or set ofprobing packets) with a corresponding TTL value set to 2. This time, thepacket will first be received by node L and the TTL value is reducedto 1. The packet then will be forwarded to node A and at that pointbecause the TTL value will be set to zero, node A will return an ICMPTime exceeded message which contains the forwarding interface to theoriginator source node 405 via the receiving interface. As describedbefore, the process then generates more probing packets with differentflow signatures to identify the interfaces (forwarding and receivinginterfaces) of all the sibling nodes to node A. In the illustratedexample, the process identifies the receiving and forwarding interfacesof node B (which is at the same TTL level as node A is) throughgenerating such additional probing pockets.

The process 300 of FIG. 3 performs 320-330 recursively until all thesubsets of receiving and forwarding interfaces of all the nodes alongdifferent paths are identified. After the process determines (at 330)that all the intermediate nodes before the destination node are probedand it has reached the destination node, the process 300 resets andreinitiates (at 350) the time to live (TTL) value in the probingpacket's header and routes the probing packet from the destination nodeback to the next forwarding element towards the source node. In otherwords, the process performs a full mesh probing on all forwardingelements that are between the source node and the destination node inorder to identify a full mesh network topology between the two nodes.That is, the process forwards new sets of probing packets to allintermediate nodes from both ways to each intermediate node in order toidentify all subsets of forwarding interfaces of the nodes (e.g.,intermediate NEFEs).

The full mesh probing of some embodiments discovers different subsets ofinterfaces of the forwarding elements. These subsets of interfaces arethen used to (1) aggregate each forwarding element's interfaces when twosubsets have intersection (i.e., common elements between the twosubsets) and (2) assert a real link between two forwarding elements whentheir forwarding interfaces point to each other (i.e., when theforwarding interfaces of the first forwarding element point to thesecond forwarding element and the forwarding interfaces of the secondforwarding element point to the first forwarding element).

Similar to identifying subsets of interfaces of each node at aparticular TTL level from the source node to the destination node, theprocess 300 forwards (at 360) new probing packets to all sibling nodesat each TTL level in order to identify forwarding and receivinginterfaces of each node at that TTL level. However, this time, becausethe process has already discovered a subset of forwarding interfaces ofthe nodes that are placed on the next TTL level, the process canrecognize which two intermediate nodes' forwarding elements are pointingto each other.

Therefore, the process 300 draws (at 370) a link between each pair ofnodes that are at neighboring TTL levels and that a subset of eachnode's forwarding interfaces point to the other node. The process thendetermines (at 380) whether all nodes on different paths are probed. Ifthe process determines that it has reached the source node (i.e.,transport node 405) the process 300 ends, otherwise the processincreases the TTL value by one (at 390) and returns to 360 which wasdescribed above.

In the illustrated example of FIG. 4, the process starts from thedestination node forwards probing packets to node E. After identifyingthat a subset of forwarding interfaces of node E at TTL level one ispointing to node C at TTL level two, while a subset of forwardinginterfaces of node C is pointing to node E, the process draws a linkbetween these two nodes. Similarly, after identifying that a subset offorwarding interfaces of node E at TTL level one is pointing to node Dat TTL level two, while a subset of forwarding interfaces of node D ispointing to node E, the process draws a link between these two nodestoo. However, and unlike other traceroute algorithms, the process willnot draw a link 450 between node A and node D because the forwardinginterfaces of these two nodes are not pointing to each other. Nor dosethe process draw a link 460 between node C and node B, since none of theforwarding elements of any of these two nodes points to the other node.

Some embodiments perform variations on the process 300. The specificoperations of the process 300 may not be performed in the exact ordershown and described. For instance, the process of some embodiments doesnot send different probing packets to identify all sibling nodes at aparticular TTL level before moving on to the next TTL level. Instead, insome embodiments, the process first discovers all the intermediate nodesalong a particular path between the pair of endpoints by forwarding theprobing packets to the same path first and after all the nodes on thepath are discovered the process starts discovering other paths betweenthe endpoints. Also, the specific operations of the process may not beperformed in one continuous series of operations, and different specificoperations may be performed in different embodiments.

II. Bandwidth Estimation

After identifying the network topology that includes all of the existingrouting paths between a pair of endpoints (e.g., a source endpoint and adestination endpoint in a software-defined datacenter) and all of theforwarding elements along each routing path, the method of someembodiments chooses a single flow signature for each identified path.The method generates bandwidth estimation probing packets with each flowsignature (e.g., having a different transmission control protocol (TCP)or user datagram protocol (UDP) source port) in order to calculate anestimated bandwidth for each link (e.g., hop-by-hop link) of the pathcorresponding to the flow signature. Some embodiments employ theenhanced traceroute algorithm that was described above in Section I toidentify the flow signatures between the pair of endpoints. A moredetailed example for identifying different flow signatures is describedbelow later in the section. The bandwidth estimation probing packets insome embodiments are generated by different bandwidth estimation toolssuch as pathchar, clink, pchar, etc.

The method of some embodiments aggregates all of the calculatedestimated bandwidths of the links in order to estimate the bandwidthbetween the two endpoints. To do so, the method normalizes the estimatedbandwidths of the diverged links (i.e., links that share the samestarting node and the same ending node). That is, the method of someembodiments first identifies each pair of nodes (i.e., diverged nodes)that shares two or more sets of diverged links. Each set of divergedlinks includes two or more successive links on a particular path thatconnect the pair of diverged nodes. The method then recursivelycalculates an estimated bandwidth for a hypothetical link between eachpair of diverged nodes. The estimated bandwidth for each hypotheticallink is calculated by (1) identifying a minimum of all estimatedbandwidths of all diverged links in each set of diverged links on aparticular path and (2) summing up the identified minimum bandwidths.After calculating the bandwidths for all hypothetical links between thediverged nodes, the method of some embodiments calculates a minimum ofestimated bandwidths of all the hypothetical links and estimatedbandwidths of all other non-diverged links that lie between theendpoints (i.e., the source endpoint and the destination endpoint) inorder to estimate the bandwidth between the two endpoints.

FIGS. 5A-5C illustrate an example of how the method of some embodimentsidentifies different flow signatures for different paths between a pairof endpoints. More specifically, FIGS. 5A-5C illustrate, through sixdifferent stages 505-530 of a network topology 500, how the method ofsome embodiments identifies the different paths between a pair ofendpoints (1) using different flow ids representing different probingpackets that are routed through different paths and (2) choosing asingle flow signature for each identified path among all the flow idsrepresenting different probing packets that are routed through thatpath.

The first stage 505 of FIG. 5A illustrates a network topology 500 with asource node (e.g., source EFE) 540 and a destination node (e.g.,destination EFE) 550. Between the nodes 540 and 550, there are severalother intermediate nodes (e.g., NEFEs) L and A-H. The first stage 505also shows that six different flow ids 1-6 representing six differentprobing packets that are forwarded through node L at TTL level one(i.e., TTL value set to one) are assigned to node L. In some embodimentseach forwarding element needs to initially have six flow ids to proceedto discover its next hops (i.e., the forwarding elements at the next TTLlevel, to which the forwarding element can forward packets).

The second stage 510 illustrates that at TTL level two, the methoddiscovers three of the flow ids (i.e., flow ids 1, 3, and 5) on node A,while the other three flow ids (i.e., flow ids 2, 4, and 6) areforwarded to node B (assuming that the flows are evenly load balanced).The third stage 515 in FIG. 5B shows that three more flow ids, namely,flow ids 7, 9, and 11 representing three more probing packets that aregenerated and forwarded through node A, are assigned to this node inaddition to the first three flow ids that were initially discovered onnode A. As described above, in some embodiments, when there can be morethan one next hop associated with a particular node, the method of someembodiments ensures that there are enough packets that can be forwardedthrough the particular node (e.g., at least six different flowsignatures in the illustrated example should be forwarded through nodeA). Here, two paths AC and AD are associated with node A while there wasinitially only three flow ids were discovered in node A. Therefore themethod has generated more packets until there are three more new flowids discovered on node A (i.e., flow ids 7, 9, and 11). Assuming thatnode L evenly balances the flows to node A and node B, the method alsodiscovers three more flow ids (i.e., flow ids 8, 10, and 12) on node Bthat is associated with the same TTL level with which node A isassociated.

As it can be noticed in the illustrated example, the newly added flowids to nodes A and B are not shown to be discovered on node L (i.e., arenot added to the previously shown flow ids for this node) even thoughthe packets that they are representing have been forwarded through nodeL. This is only for simplifying the description of the figure. Also, oneof ordinary skill in the art would realize that the illustrated flow idsare for exemplary purposes and each node in reality can be assigned moreor less number of flow ids. For instance, after flow ids 1, 3, and 5,were initially discovered on node A in this example, six more flow idsare generated (i.e., flow ids 7-12), three of which are added to flowids of node A and the other three are assigned to node B based on theassumption that node L evenly balanced the flows to node A and node B.In reality, however, it is possible that many more probing packets aregenerated to fill the gap for node A and the flow ids in the example arepresented only for the simplicity of the explanation.

The fourth stage 520 illustrates that similarly, the method hasdiscovered three flow ids 1, 5, and 9 initially on node C while threeflow ids 3, 7, and 11 are discovered on node D. For the same reason thatdiscussed above, this stage also shows the method has generated moreprobing packets for nodes C and D to make sure they have enough numberof flow ids forwarded to them and subsequently assigned additional flowids 13, 17, and 21 to node C and flow ids 15, 19, and 23 to node D. Thefourth stage 520 also shows that node E which is also associated withTTL level 3, has been assigned additional flow ids 14, 16, 18, 20, and22 on top of the flow ids 2, 4, 6, 8, 10, and 12 that were originallydiscovered on this node.

The fifth stage 525 in FIG. 5C shows that at the next TTL level (i.e.,TTL level 4) all the flow ids that were assigned to nodes C and D in theprevious TTL level are combined and assigned to node F which is thereceiving node associated with nodes C and D. Similarly, this stageshows that all the flow ids that were assigned to node E are nowassigned to node G as well. Since there were enough flow ids to coverthe next potential paths associated with the nodes at this TTL level, nomore probing packets are generated and therefore no additional flow idsare assigned to any of the nodes at this TTL level.

Finally, the sixth stage 530 illustrates that at the last TTL level 5,all the flow ids that were assigned to nodes F and G at the previous TTLlevel, are now combined and assigned to node H since this node is thereceiving node associated with nodes F and G. Therefore as shown, node Hhas been assigned all flow ids 1-23. After assigning the flow ids to allthe nodes between a pair of endpoints, the method of some embodimentschooses a single flow signature for each path, where the packetgenerated with that flow signature can be routed from the source node tothe destination node through all the intermediate nodes along the path.

FIG. 6 illustrates an example of choosing a single flow signature foreach identified path among all the flow ids representing differentprobing packets that are routed through that path. More specifically,this figure shows at stage 610 that the method has chosen flow signature21 representing one of the probing packets that was routed through thehighlighted path L-A-C-F-H from the source node to the destination node.

The next stage 620 shows that the method has chosen flow signature 23that represents one of the probing packets that is forwarded through thehighlighted path L-A-D-F-H from the source node to the destination node.Similarly, at stage 630, the method has chosen flow signature 22 thatrepresents one of the probing packets that is routed through thehighlighted path L-B-E-G-H from the source node to the destination node.One of ordinary skill in the art would realize that the illustratedfigure and network topology is given as an example and there could bemany more intermediate nodes in a network topology that are connected inmany other ways. However, the method of assigning flow ids and choosingflow signatures for all other network topologies will be essentially thesame as the method that was described above.

FIG. 7 conceptually illustrates a process 700 of some embodiments thatcalculates a bandwidth estimation for each link (e.g., hop-by-hop link)that is along the different routing paths between a pair of endpoints.In some embodiments the process 700 is implemented by a bandwidthmeasurement agent or tool on a transport node at the edge of the network(as will be discussed in more detail in Section III below). FIG. 7 willbe described by reference to FIG. 8, which illustrates an example of howthe process 700 calculates an estimated bandwidth for each link alongthe different paths.

The process 700 begins by selecting (at 710) the first flow signaturethat the method has identified for the first routing path between theendpoints. The process 700 then generates (at 720) and routes bandwidthestimation probing packets with the selected flow signature through therouting path that corresponds to the selected flow signature. In someembodiments, the bandwidth estimation probing packets are generated withthe selected flow signature by encapsulating the forwarding address ofthe flow signature in the probing packets' headers. In this manner, theprobing packets will be forwarded through the same identified pathtowards the destination endpoint that corresponds with the selected flowsignature. In some embodiments, the bandwidth estimation probing packetsare generated with the flow signature by different bandwidth estimationtools such as pathchar, clink, pchar, etc.

Next, the process 700 calculates (at 730) the estimated bandwidth foreach link that is along the selected routing path. In some embodiments,the process tries to infer the bandwidth (and/or latency) of individuallinks along the path by measuring the round trip time of probing packetssent from the source node. The process of some embodiments takesadvantage of the TTL field in the probing packet. As described above,the value set in the TTL field determines how many links a packet cantraverse before it expires. If a forwarding element receives a probingpacket that has expired, it drops the packet and sends an ICMP errorpacket back to the sender. The source address of the error packetindicates which forwarding element (e.g., router) the outgoing packetreached before expiring. The process of some embodiments sends out aseries of probing packets with varying values of TTL field and varyingpacket sizes. For each probing packet, the process measures the timeuntil the error packet is received. By performing statistical analysisof these measurements, the process of some embodiments infers thebandwidth (and/or latency) of each link on the path.

FIG. 8 illustrates a network topology 800 which is similar to thenetwork topology that was described by reference to FIGS. 5A-5C and FIG.6. FIG. 8 shows that the bandwidth estimation method has identified theestimated bandwidths of the links along the three identified routingpaths between the source node and the destination node (e.g., throughthe method that is described above). The first identified path in thefigure as described by reference to FIG. 6 corresponds to flow signature21 and includes the nodes Src, L, A, C, F, H, and Dst. The method hassent bandwidth estimation probing packets generated by an estimationtool to estimate the bandwidth of all the links on this path. Thesebandwidth estimations are illustrated as B(Src, L) which is thebandwidth estimation for the link between the source node and node L,B(L, A) which is the bandwidth estimation for the link between node Land node A, B(A, C) which is the bandwidth estimation for the linkbetween node A and node C, B(C, F) which is the bandwidth estimation forthe link between node C and node F, B(F, H) which is the bandwidthestimation for the link between node F and node H, and finally B(H, Dst)which is the bandwidth estimation for the link between node H and thedestination node.

The process 700 then determines (at 740) whether there are more flowsignatures to select. If the process 700 determines that the probed flowsignature was the last flow signature and there are no more flowsignatures to select, the process ends. On the other hand, if theprocess 700 determines that more flow signatures are left to probe, theprocess selects the next flow signature and returns to 720 which wasdescribed above.

FIG. 8 shows that the bandwidth estimation method has selected thesecond flow signature 23 corresponding to the identified path thatincludes the nodes Src, L, A, D, F, H, and Dst. The method has sentprobing packets with an estimation tool to estimate the bandwidth of allthe links on this path. These bandwidth estimations are illustrated asB(Src, L) which is the bandwidth estimation for the link between thesource node and node L, B(L, A) which is the bandwidth estimation forthe link between node L and node A, B(A, D) which is the bandwidthestimation for the link between node A and node D, B(D, F) which is thebandwidth estimation for the link between node D and node F, B(F, H)which is the bandwidth estimation for the link between node F and nodeH, and finally B(H, Dst) which is the bandwidth estimation for the linkbetween node H and the destination node.

Lastly, FIG. 8 also shows that the method has selected the third andlast flow signature 22 corresponding to the identified path thatincludes the nodes Src, L, B, E, G, H, and Dst. The method has sentprobing packets with an estimation tool to estimate the bandwidth of allthe links on this path. These bandwidth estimations are illustrated asB(Src, L) which is the bandwidth estimation for the link between thesource node and node L, B(L, B) which is the bandwidth estimation forthe link between node L and node B, B(B, E) which is the bandwidthestimation for the link between node B and node E, B(E, G) which is thebandwidth estimation for the link between node E and node G, B(G, H)which is the bandwidth estimation for the link between node G and nodeH, and finally B(H, Dst) which is the bandwidth estimation for the linkbetween node H and the destination node.

In some embodiments, the process calculates the bandwidth capacity ofeach link and path when it calculates the estimated bandwidth for thelinks and paths. In some embodiments, the process calculates theavailable minimum bandwidth of the network traffic (hop-by-hop links andend-to-end path). In some embodiments, the process calculates bothcapacity and available bandwidth when it estimates the bandwidth of eachlink and path in the network topology.

Some embodiments perform variations on the process 700. The specificoperations of the process 700 may not be performed in the exact ordershown and described. The specific operations may not be performed in onecontinuous series of operations, and different specific operations maybe performed in different embodiments.

As described above, after calculating an estimated bandwidth for eachlink that is between the pair of endpoints, the method of someembodiments aggregates all of the calculated estimated bandwidths of thelinks in order to estimate the bandwidth between the two endpoints. Inorder to aggregate, the method of some embodiments normalizes theestimated bandwidths of the diverged links first and then calculates aminimum bandwidth of the normalized links and the non-diverged linksbetween the endpoints to complete the aggregation.

In order to normalize, the method of some embodiments first identifiesthe diverged nodes. The method then recursively calculates an estimatedbandwidth for a hypothetical link between each pair of diverged nodes.After calculating the bandwidths for all the diverged nodes, the methodof some embodiments calculates a minimum of estimated bandwidths of allthe hypothetical links and estimated bandwidths of all othernon-diverged links that lie between the endpoints in order to estimatethe bandwidth between the two endpoints.

FIG. 9 conceptually illustrates a process 900 of some embodiments thataggregates the calculated bandwidths of all the links between the pairof endpoints to generate an estimated bandwidth between the endpoints.FIG. 9 will be described by reference to FIG. 10 which illustrates anexample of how the process 900 aggregates the estimated bandwidths ofthe links by (1) normalizing the bandwidths between each pair ofdiverged nodes and (2) calculating a minimum bandwidth of the linksbetween diverged nodes and non-diverged nodes.

The process 900 begins by receiving (at 910) a network topology betweena pair of endpoints. The network topology, as described above, isgenerated in some embodiments by a network measurement service runningon a network manager. The network measurement service of someembodiments facilitates in generating a topology between a pair ofendpoints (e.g., a pair of transport nodes) by communicating with a pairof network measurement agents that operate on the pair of endpoints andcause the endpoints forward probing packets to each other (e.g., a fullmesh probing that was discussed in Section I).

Next, the process 900 selects (at 920) the first link on the pathbetween the source node and the destination node. FIG. 10 shows the samenetwork topology 800 that is illustrated in FIG. 8. The first linkbetween the source node and the destination node in this networktopology is the link between the source node and node L, therefore theprocess selects this node.

The process 900 then determines (at 930) whether the selected link is adiverged link in a set of diverged links. As described above, divergedlinks are the links that share the same starting and ending nodes(diverged nodes). If the selected link is not a diverged link in a setof diverged links, the process proceeds to 950, which will be describedlater below. As shown in FIG. 10, the first selected link, which is thelink between the source node and node L, does not share these nodes withany other link and therefore this link is not a diverged link.

On the other hand, if the process 900 determines (at 930) that theselected link is a diverged link (i.e., a link that shares theforwarding node with at least one other link in the network topology),the process normalizes the bandwidths of all the diverged links betweeneach pair of diverged nodes that are situated on either side of thediverged links. In order to normalize, the process of some embodimentsfirst identifies each pair of diverged nodes. Each set of diverged linksincludes two or more successive links on a particular path that connectsthe pair of diverged nodes.

The method then recursively calculates an estimated bandwidth for ahypothetical link between each pair of diverged nodes. That is, whenthere are nested pairs of diverged nodes inside other pairs of divergednodes, the process starts with the most inner pair of diverged nodes andcontinues the calculations recursively outward. The estimated bandwidthfor each hypothetical link is calculated by (1) identifying a minimum ofall estimated bandwidths of all diverged links in each set of divergedlinks on a particular path and (2) summing up the identified minimumbandwidths.

After the normalization, the process 900 determines (at 950) whetherthere are more links to process. If there are more links, the processreturns to 920 to select the next link and to decide whether tonormalize that link too or not. If the process determines (at 950) thatno more links left, the process will proceed to 960, which will bedescribed below.

Returning to FIG. 10, as explained above, the first link is not adiverged link, therefore the process selects the next link on the pathwithout performing a normalization. The next link (i.e., the linkbetween node L and node A) however, is a diverged link because it sharesthe forwarding node L with another link (i.e., link LB). Therefore, andas described above, the process recursively normalizes the divergedlinks starting from the most inner pair of diverged nodes. In theillustrated network topology, the most inner pair of diverged nodes isthe pair of nodes A and F. Therefore, the process starts by normalizingthe diverged links between diverged nodes A and F.

In order to normalize the diverged links and calculate an estimatedbandwidth between the diverged nodes (e.g., bandwidth estimation for thehypothetical link AF), the process of some embodiments first identifiesa minimum of all estimated bandwidths of all diverged links in each setof diverged links on a particular path between the diverged nodes. Asshown in FIG. 10 the diverged links in the first set of diverged linksbetween nodes A and F are the links AC and CF that are along the firstpath and the diverged links in the second set of diverged links are thelinks AD and DF that are along the second path (e.g., the first andsecond paths identified by the method as described above by reference toFIG. 6).

After calculating the minimum bandwidths for each set of diverged links,the process adds up the minimum bandwidth of each set together togenerate the estimated bandwidth for the hypothetical link AF.Therefore, in FIG. 10, the estimated bandwidth between the divergednodes A and F (i.e., B(A, F)) will be the minimum bandwidth of thediverged links LC and CF (i.e., MIN(B(L, C), B(C, F))) plus the minimumbandwidth of the diverged links AD and DF (i.e., MIN(B(A, D), B(D, F)))as shown in the figure.

The process 900 after calculating the bandwidth between the divergednodes A and F will then start calculating the bandwidth between theouter diverged nodes L and H by normalizing the diverged links betweenthese nodes. That is, the process first identifies the different sets ofdiverged links between the two nodes and then adds up the minimum ofeach identified set to the other sets. Therefore, the estimatedbandwidth between the nodes L and H (for the hypothetical link LH) willbe determined by calculating a minimum of the bandwidths of the firstset of diverged links which includes the links LA, AF, and FH and thesecond set of diverged links which includes the links LB, BE, EG, andGH; and then adding the two minimum variables. The estimated bandwidthfor link LH will therefore be MIN(B(L, A), B(A, F), B(F, H)) plusMIN(B(L, B), B(B, E), B(E, G), B(G, H)).

After the normalization of these diverged links, the process determinesthat one more link is left between the pair of endpoints, which is thelink between node H and the destination endpoint. Since this link is nota diverged link the process does not perform the normalization process.Also since this link is the last link in the network topology, theprocess proceeds to 960. At 960, the process calculates a minimumbandwidth of all links between the diverged nodes and non-diverged nodesas the estimated bandwidth between the pair of endpoints. The processthen ends.

In FIG. 10, the process has determined that the link between the sourcenode and node L and the link between node H and the destination node arenot diverged links. The process has also calculated an estimatedbandwidth for all the diverged links between node L and node H asdescribed above. Therefore, the process can now calculate an estimatedbandwidth between the pair of endpoints by calculating a minimum ofthese estimated bandwidths. Therefore, the estimated bandwidth betweenthe pair of endpoints will be MIN(B(Src, L), B(L, H), B(H, Dst)).

Some embodiments perform variations on the process 900. The specificoperations of the process 900 may not be performed in the exact ordershown and described. For example, in some embodiments, the processstarts by selecting the nodes instead of the links. That is, the processafter selecting each node, determines whether the node is a divergednode and then performs the normalization based on that determination.The specific operations may not be performed in one continuous series ofoperations, and different specific operations may be performed indifferent embodiments.

III. Periodical Probing

In some embodiments, the method generates different network measurementsdata (e.g., network topology, bandwidth estimation of different paths,etc.) for a pair of endpoints upon receiving a network administrativerequest (e.g., an application programming interface (API) request) or byother means (e.g., automatically and without intervention of a user). Insome embodiments, the method is implemented by a network measurementagent operating on each endpoint and a centralized service component(e.g., web service layer) executing on a network manager machine (e.g.,a controller) that responds to measurement requests. The networkmeasurement agent of some embodiments probes the network periodically(i.e., in certain time intervals) or upon an API request that itreceives through the network manager machine to gather the measurementdata of a particular network topology between a pair of endpoints (e.g.,the pair of endpoints on which the network measurement agents operate).

In some embodiments, when a network measurement feature is turned “on”or enabled (e.g., through an API), the network measurement agent canprovide (e.g., to a network measurement server or the network managermachine) real-time data (e.g., bandwidth capacity, available bandwidth,etc.) or the most recent measurement data on each possible path (e.g., alink, multiple links, etc.) between a given pair of internet protocol(IP) endpoints. The endpoints of some embodiments are transport nodesthat belongs to different hosts of the network. Each host of the networkin some embodiments is a computing machine that hosts several virtualmachines (VMs). Each VM of a host communicates with other VMs of thesame host or other hosts in the network through the transport nodes(e.g., hardware or software-defined EFEs) of host(s).

For instance, with an API request via a network measurement interface ofa network manager machine, the network measurement agent of someembodiments can periodically measure the bandwidths between a pair ofendpoints of the network. This is important because the data (e.g.,real-time data) can be used in a number of different ways, includingforwarding traffic along different paths. For instance, the always “on”feature can be used to retrieve real-time measurement data and assigndifferent flow signatures or hashes for different data flows dependingon the measurement data. With the hop-by-hop measurements and thedifferent flow signatures associated with a given topology, the flow ofdata, from a source to a destination, can even be controlled on a perhop basis. The network measurement server of some embodiments stores theprovided network measurement data in a storage (e.g., a database).Rather than simply storing the measurement data, the server of someembodiments accumulates or aggregates the measurement data. That is, fora pair of endpoints, the server stores the measurement data and thenupdates the data with each updated measurement data it receives from themeasurement agents.

While the embodiments described above and below provide a networkmeasurement agent for estimating bandwidth for a pair of endpoints orgenerating network topology between the endpoints, many embodiments ofthe invention are equally applicable to a combination of a networkmeasurement interface, as described above, and a set of one more networkmeasurement agents that operate on different transport nodes. In otherwords, the operations described above and below that are done by anetwork measurement agent are equally achievable, in some embodiments,by a network measurement interface that runs on a network managermachine (e.g., a controller) and its corresponding set of networkmeasurement agents that operate on different transport nodes.

FIG. 11 presents a state diagram 1100 that illustrates exampleoperations performed by such a network measurement agent. This figureshows several states 1110-1160 that collectively perform theseoperations. Each of the states represents a set of one or more processesfor performing a subset of the operations. The states include a loadingstate 1110, a waiting state 1120, a probing network state 1130, a gettopology state 1140, a get bandwidth state 1150, and an updatemeasurement storage state 1160. Not all operations have to be performedby the network measurement agent. For instance, the network measurementagent of some embodiments does not update the measurement data in themeasurement storage and instead, the service component or another moduleperforms the storage updating operation. Also not all transitionsbetween the operations are automatic. For instance, as stated above, anetwork measurement agent might probe the network upon a user request(e.g., API request) and not necessarily after a time interval isexpired.

In the load measurement state 1110, the network measurement agent isuploaded on the forwarding element. In some embodiments, the measurementagent is a software agent that is uploaded onto the edge forwardingelement of a host machine of a datacenter each time the edge forwardingelement is instantiated in the network (e.g., each time the host machineis connected to the network for the first time). In some embodiments themeasurement agent is uploaded on a hypervisor of the host machine. Insome embodiments the measurement agent is uploaded on a virtual networkinterface card of a source VM or a destination VM that runs on a hostmachine.

In the waiting state 1120, the network measurement agent of someembodiments waits for a certain time interval to expire in order tostart probing a network (e.g., the network topology between twoendpoints). Therefore, the network measurement agent periodically (i.e.,when the time interval has expired) and without any user (e.g., networkadministrator) intervention transitions to the probing network state1130. In the probing network state, the network measurement agent startsprobing the network by first determining whether the network topologyhas changed or not. The measurement agent transitions to the gettopology state to make such determination.

As stated before, the network topology includes a set of physicalforwarding element (hardware or software-defined forwarding elements) aswell as a set of logical forwarding elements. The set of physicalforwarding elements of some embodiments can be changed in a network(e.g., a software-defined datacenter network) rapidly as forwardingelements (hardware or software-defined) can be added to, removed from,or transferred to a different host machine of the network or a hostmachine of a different network (e.g., another software-defineddatacenter network). These changes would result in changes of networktopology between two endpoints.

In the get topology state, the network measurement agent of someembodiments identifies a network topology that is involved in routing ofthe network traffic between the edge forwarding element on which thenetwork agent runs and another edge forwarding element to which thenetwork traffic is routed. The network measurement agent of someembodiments identifies the network topology using the enhancedtraceroute method that was described above in Section I and by referenceto FIGS. 3 and 4. If the network measurement agent determines that thereis no change in the network topology, in some embodiments, the agenttransitions back to the waiting state 1120.

However, in some embodiments, even when there is no change in thenetwork topology, the network measurement agent transitions from probingnetwork state 1130 to get bandwidth state 1150 in order to calculate theavailable minimum bandwidth of the network traffic (hop-by-hop links andend-to-end path) in spite of no change in the estimated capacity of thenetwork. When there is a change in the network topology, the networkmeasurement agent of some embodiments transitions to get bandwidthstate.

In the get bandwidth state 1160, the network measurement agentcalculates the estimated bandwidths (e.g., available bandwidth andbandwidth capacity) of all the existing links and paths in the networktopology. That is, the agent calculates the bandwidths of the links andpaths between the edge forwarding element on which the network agentruns and another edge forwarding element to which the network traffic isrouted. Several examples of calculating the estimated bandwidth havebeen described above in Section II by reference to FIGS. 5-10. Aftercalculating the bandwidth estimation, the measurement agent transitionsto the update measurement storage state 1160.

In the update measurement storage state 1160, the network measurementagent of some embodiments stores the network measurement data in astorage (e.g., a database) so later the forwarding element can retrievethese data and make decision regarding which path the network trafficshould be routed. Rather than simply storing the measurement data, theedge forwarding element of some embodiments accumulates or aggregatesthe measurement data. That is, for a pair of endpoints, the forwardingelement stores the measurement data and then updates the data with eachupdated measurement data it receives from the measurement agent.

As described before, by knowing the estimated bandwidth for differentpaths, a controller that interacts with a pair of forwarding elements ofa network can decide which path to use for packets that are associatedwith different data flows (e.g., elephant flows, mouse flows, etc.). Forexample in some embodiments, the network manager can decide to route adata flow from a first endpoint device to a second endpoint devicethrough a particular routing path when an estimated bandwidth of theparticular routing path is more than a threshold amount or less than athreshold amount.

In this manner, one set of packets belonging to a particular data flowis not held behind a routing path that is used by another set of packetsbelonging to another data flow for a prolonged period of time.Alternatively, or conjunctively, the forwarding element can decide toswitch the transportation of a particular data flow from one path toanother path based on the capacity and available bandwidth of each path.In some embodiments, an edge forwarding element breaks an elephant flowto multiple mice flows and route each mouse flow to a different routingpath based on the capacity and available bandwidth that it receives fromits corresponding network measurement agent. The forwarding elements ofsome embodiments make such decisions through one or more controllers(e.g., hardware of software-defined controllers) that control theforwarding elements.

FIG. 12 illustrates an example of how the method of some embodimentsroutes different sets of packets that belong to different data flowsthrough different paths between two endpoints based on the estimatedbandwidth of each path. More specifically FIG. 12 shows a source node1210 and a destination node 1220 and several intermediate nodes (e.g.,NEFEs) along three different paths between the pair of endpoints. Thisfigure also shows A network manager 1240 (e.g., a controller) which insome embodiments is used to manage and configure the endpoints 1210 and1220. The network manager 1240 of some embodiments includes ameasurement interface (not shown) that communicates with the networkmeasurement agents 1230 that operate on the transport nodes (i.e., endpoints) 1210 and 1220.

The network manager 1240 in some embodiments perform additionalmanagement services by sending (e.g., pushing) management data (e.g.,bridge information, virtual interface information, tunnel information,etc.) to the physical forwarding elements (e.g., transport nodes). Thenetwork manager of some embodiments performs the configuration bysending (e.g., pushing) data flows to the transport nodes. Instead ofsending the data flows, the network manager 1240 of some embodimentsexchanges forwarding state information that is translated into a set ofdata flows at the transport nodes.

FIG. 12 shows that the network manager 1240 and its correspondingnetwork measurement agents 1230 have already recognized the networktopology between the pair of endpoints 1210 and 1220. The figure alsoshows that these measurement tools have identified three different pathsbetween the pair of endpoints with three different bandwidths. Morespecifically, the path LADG has a higher bandwidth than the path LBEGwhereas the path LBEG has a higher bandwidth that the path LCFG. Thedifference in the bandwidth in the figure is shown by differentthickness for each path.

By recognizing the different paths and the different bandwidths for eachpath, the method of some embodiments can handle packets that belong todifferent data flows differently. For example, in some embodiments atransport node at the edge of the network (i.e., an endpoint) or itscorresponding controller monitors statistics or measurements relating toa data flow. As an example, the transport node may track the number ofbytes sent in the data flow, and specify, based on the number of bytes,whether the data flow should be classified as an elephant flow or amouse flow.

Once an elephant flow is detected, the transport node of someembodiments treats it differently than a mouse flow. For example thenetwork manager 1240 of some embodiments forwards the packets that areassociated with the elephant flow through a path with a lower bandwidth(e.g., path LCFG) between the endpoints. This is because an elephantflow typically represents a long-lived data flow that is associated withhigh volume connection and therefore tends to fill network paths whichin turn introduces non-trivial queuing delay to any other data flow thatshares the paths.

Different from an elephant flow, a mouse flow represents a short-livedflow. Mice are often associated with bursty, latency-sensitiveapplications, whereas elephants tend to be associated with large datatransfers in which throughput is far more important than latency.Therefore the network manager 1240 of some embodiments forwards thepackets that are associated with the mouse flow through a path with ahigher bandwidth (e.g., path LADG) between the endpoints. In thismanner, the set of packets belonging to the mouse flow is not heldbehind the path that is used by the set of packets that belong to themouse flow for a prolonged period of time. Some embodiments that detectand handle elephant flows are described in U.S. patent application Ser.No. 14/231,647, entitled “Detecting and Handling Elephant Flows,” filedMar. 31, 2014, which is incorporated herein by reference.

IV. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or morecomputational or processing unit(s) (e.g., one or more processors, coresof processors, or other processing units), they cause the processingunit(s) to perform the actions indicated in the instructions. Examplesof computer readable media include, but are not limited to, CD-ROMs,flash drives, random access memory (RAM) chips, hard drives, erasableprogrammable read-only memories (EPROMs), electrically erasableprogrammable read-only memories (EEPROMs), etc. The computer readablemedia does not include carrier waves and electronic signals passingwirelessly or over wired connections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 13 conceptually illustrates an electronic system 1300 with whichsome embodiments of the invention are implemented. The electronic system1300 may be a computer (e.g., a desktop computer, personal computer,tablet computer, etc.), server, dedicated switch, phone, PDA, or anyother sort of electronic or computing device. Such an electronic systemincludes various types of computer readable media and interfaces forvarious other types of computer readable media. Electronic system 1300includes a bus 1305, processing unit(s) 1310, a system memory 1325, aread-only memory 1330, a permanent storage device 1335, input devices1340, and output devices 1345.

The bus 1305 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1300. For instance, the bus 1305 communicativelyconnects the processing unit(s) 1310 with the read-only memory 1330, thesystem memory 1325, and the permanent storage device 1335.

From these various memory units, the processing unit(s) 1310 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 1330 stores static data and instructions thatare needed by the processing unit(s) 1310 and other modules of theelectronic system. The permanent storage device 1335, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system1300 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 1335.

Other embodiments use a removable storage device (such as a floppy disk,flash memory device, etc., and its corresponding drive) as the permanentstorage device. Like the permanent storage device 1335, the systemmemory 1325 is a read-and-write memory device. However, unlike storagedevice 1335, the system memory 1325 is a volatile read-and-write memory,such a random access memory. The system memory 1325 stores some of theinstructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory1325, the permanent storage device 1335, and/or the read-only memory1330. From these various memory units, the processing unit(s) 1310retrieves instructions to execute and data to process in order toexecute the processes of some embodiments.

The bus 1305 also connects to the input and output devices 1340 and1345. The input devices 1340 enable the user to communicate informationand select commands to the electronic system. The input devices 1340include alphanumeric keyboards and pointing devices (also called “cursorcontrol devices”), cameras (e.g., webcams), microphones or similardevices for receiving voice commands, etc. The output devices 1345display images generated by the electronic system or otherwise outputdata. The output devices 1345 include printers and display devices, suchas cathode ray tubes (CRT) or liquid crystal displays (LCD), as well asspeakers or similar audio output devices. Some embodiments includedevices such as a touchscreen that function as both input and outputdevices.

Finally, as shown in FIG. 13, bus 1305 also couples electronic system1300 to a network 1365 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 1300 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself. In addition, someembodiments execute software stored in programmable logic devices(PLDs), ROM, or RAM devices.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium,” “computer readable media,” and “machinereadable medium” are entirely restricted to tangible, physical objectsthat store information in a form that is readable by a computer. Theseterms exclude any wireless signals, wired download signals, and anyother ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 1, 3, 7, and 9) conceptually illustrate processes. Thespecific operations of these processes may not be performed in the exactorder shown and described. The specific operations may not be performedin one continuous series of operations, and different specificoperations may be performed in different embodiments. Furthermore, theprocess could be implemented using several sub-processes, or as part ofa larger macro process. Thus, one of ordinary skill in the art wouldunderstand that the invention is not to be limited by the foregoingillustrative details, but rather is to be defined by the appendedclaims.

1-23. (canceled)
 24. A method for measuring an estimated bandwidthbetween two endpoints of a network, the method comprising: identifying aplurality of routing paths between the two endpoints, wherein therouting paths comprise a set of intermediate nodes and a set of linksthat each connect a pair of intermediate nodes, wherein the set of linkscomprises (i) a plurality of direct links that each directly connects apair of intermediate nodes and (ii) at least one multi-path link that isa link between a pair of intermediate nodes for which at least twodivergent sets of direct links exist in the plurality of routing paths;calculating an estimated bandwidth for each direct link in theidentified routing paths; normalizing the estimated bandwidths for eachmulti-path link; and calculating an estimated bandwidth using thenormalized bandwidths for the multi-path links and a minimum bandwidthcalculated for each remaining direct link that is not part of amulti-path link.
 25. The method of claim 24, wherein identifying theplurality of routing paths comprises forwarding probing packets throughthe routing paths to identify a network topology between the endpoints.26. The method of claim 25, wherein one of the endpoints is a sourcetransport node and the other endpoint is a destination transport node,wherein forwarding the probing packets through the routing pathscomprises causing the source transport node to send a first set ofprobing packets to the destination transport node, wherein thedestination transport further sends a second set of probing packets tothe first transport node.
 27. The method of claim 24, wherein eachintermediate node is a non-edge forwarding element (NEFE), whereinidentifying the plurality of routing paths comprises identifying a setof receiving interfaces and a set of forwarding interfaces associatedwith each NEFE.
 28. The method of claim 24 further comprising providingthe calculated estimated bandwidth to a management server.
 29. Themethod of claim 24, wherein at least a particular multi-path linkcomprises at least one sub-link that is a multi-path link. whereinnormalizing the estimated bandwidths or the particular multi-path linkcomprises: normalizing the estimated bandwidth for the multi-pathsub-links; and calculating the estimated bandwidth for the particularmulti-path link using the normalized bandwidths for the multi-pathsub-links and a minimum bandwidth calculated for each remaining directlink of the particular multi-path link.
 30. The method of claim 24,wherein each endpoint is an internet protocol (IP) transport node at anedge of a network.
 31. A non-transitory machine readable medium storinga program which when executed by at least one processing unit measuresan estimated bandwidth between two endpoints of a network, the programcomprising sets of instructions for: identifying a plurality of routingpaths between the two endpoints, wherein the routing paths comprise aset of intermediate nodes and a set of links that each connect a pair ofintermediate nodes, wherein the set of links comprises (i) a pluralityof direct links that each directly connects a pair of intermediate nodesand (ii) at least one multi-path link that is a link between a pair ofintermediate nodes for which at least two divergent sets of direct linksexist in the plurality of routing paths; calculating an estimatedbandwidth for each direct link in the identified routing paths;normalizing the estimated bandwidths for each multi-path link; andcalculating an estimated bandwidth using the normalized bandwidths forthe multi-path links and a minimum bandwidth calculated for eachremaining direct link that is not part of a multi-path link.
 32. Thenon-transitory machine readable medium of claim 31, wherein the set ofinstructions for identifying the plurality of routing paths comprises aset of instructions for forwarding probing packets through the routingpaths to identify a network topology between the endpoints.
 33. Thenon-transitory machine readable medium of claim 32, wherein one of theendpoints is a source transport node and the other endpoint is adestination transport node, wherein the set of instructions forforwarding the probing packets through the routing paths comprises a setof instructions for causing the source transport node to send a firstset of probing packets to the destination transport node, wherein thedestination transport further sends a second set of probing packets tothe first transport node.
 34. The non-transitory machine readable mediumof claim 31, wherein each intermediate node is a non-edge forwardingelement (NEFE), wherein the set of instructions for identifying theplurality of routing paths comprises a set of instructions foridentifying a set of receiving interfaces and a set of forwardinginterfaces associated with each NEFE.
 35. The non-transitory machinereadable medium of claim 31 further comprising providing the calculatedestimated bandwidth to a management server.
 36. The non-transitorymachine readable medium of claim 31, wherein at least a particularmulti-path link comprises at least one sub-link that is a multi-pathlink. wherein the set of instructions for normalizing the estimatedbandwidths or the particular multi-path link comprises sets ofinstructions for: normalizing the estimated bandwidth for the multi-pathsub-links; and calculating the estimated bandwidth for the particularmulti-path link using the normalized bandwidths for the multi-pathsub-links and a minimum bandwidth calculated for each remaining directlink of the particular multi-path link.
 37. The non-transitory machinereadable medium of claim 31, wherein each endpoint is an internetprotocol (IP) transport node at an edge of a network.
 38. Anon-transitory machine readable medium storing a program that whenexecuted by at least one processing unit identifies a network topologybetween two endpoints of a network, the program comprising sets ofinstructions for: forwarding a plurality of probing packets through aplurality of routing paths that are between the two endpoints;identifying a plurality of intermediate nodes between the two endpointsusing the forwarded packets, each intermediate node identified by a setof receiving interfaces and a set of forwarding interfaces associatedwith the intermediate node; and identifying the network topology byspecifying links between different pairs of intermediate nodes using thesets of receiving and forwarding interfaces associated with eachintermediate node.
 39. The non-transitory machine readable medium ofclaim 38, wherein each endpoint is an internet protocol (IP) transportnode at an edge of the network.
 40. The non-transitory machine readablemedium of claim 38, wherein each intermediate node is a non-edgeforwarding element (NEFE).
 41. The non-transitory machine readablemedium of claim 38, wherein one of the two endpoints is a sourcetransport node while the other endpoint is a destination transport node.42. The non-transitory machine readable medium of claim 41, wherein theset of instructions for forwarding the plurality of probing packetscomprises a set of instructions for causing the source transport nodeforward a first set of probing packets to the destination transport nodeand causing the destination transport node forward a second set ofprobing packets to the source transport node.
 43. The non-transitorymachine readable medium of claim 38, wherein the intermediate nodes areat different time to live (TTL) levels between the two endpoints.